Repository navigation
Conversation
Quote non-finite FLOAT and DOUBLE leaves in the shared JSON formatter instead of aborting row iteration. Preserve finite formatting, SQL NULL, top-level numeric types, and existing kernel MAP-key spellings. Add cross-backend regression coverage and document the encoding. Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com> Co-authored-by: Isaac <no-reply@databricks.com>
| func BenchmarkAppendScalars(b *testing.B) { | ||
| // The last three take json.Marshal's fallback. | ||
| // The escaped string and []byte take json.Marshal's fallback. | ||
| values := []any{int32(42), int64(-9000), 3.14, 0.30000000000000004, float32(0.5), true, "alpha0", "café crème", |
There was a problem hiding this comment.
math.NaN() is still in BenchmarkAppendScalars. It's now a fast-path append for jsonfmt but an error path for the encoding_json baseline, so the comparison is skewed. Could we drop it, or add a comment that the baseline errors on it?
| if f < 0 { | ||
| return "-Inf" | ||
| } | ||
| return "+Inf" |
There was a problem hiding this comment.
Could we remove MapKey and use "NaN", "Infinity", and "-Infinity" consistently for both keys and values? Currently, a map containing positive infinity as both key and value renders as {"+Inf":"Infinity"}.
JDBC’s text rendering and Node’s object keys use Infinity/-Infinity; Python preserves native numeric keys. The +Inf/-Inf spelling comes from the kernel’s old formatting fallback, while Thrift previously errored on these keys. Using the same spelling throughout would simplify the contract; we can document the kernel key spelling change explicitly in the changelog.
Summary
Fixes #573.
Valid SQL FLOAT and DOUBLE values containing NaN or infinities currently abort row iteration when nested inside ARRAY, MAP, or STRUCT results. The Go renderers serialize scalar leaves through
encoding/json, which rejects non-finite floats.Render these leaves as the JSON strings
"NaN","Infinity", and"-Infinity"in the shared formatter used by both Thrift and kernel-backed results. Keep SQL NULL asnullrather than losing the distinction between special floats and missing values.Compatibility
"NaN","+Inf", and"-Inf". Thrift now accepts these previously rejected keys with the same spelling.anyor a custom decoder rather than plain float fields; this is documented in README and package docs.Test Plan
CGO_ENABLED=0 go test ./... -count=1passed with live warehouse credentials removed from the test environment.CGO_ENABLED=1 go test -race ./internal/jsonfmt ./internal/arrowscan ./internal/rows/arrowbased -count=1passed.CGO_ENABLED=0 go vet ./...passed.make lintpassed with zero issues.No live warehouse tests or native-kernel-linked E2E tests were run. Both Go renderers are exercised directly with in-memory Arrow arrays.
This pull request and its description were written by Isaac.